Voice integration platform

ABSTRACT

A voice integration platform and method provide for integration of a voice interface with a data system that includes stored data. The voice integration platform comprises one or more generic software components, the generic software components being configured to enable development of a specific voice user interface that is designed to interact with the data system in order to present the stored data to a user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.11/189,990, filed Jul. 27, 2005, which is a continuation of U.S.application Ser. No. 09/855,004, entitled “Voice Integration Platform,”filed May 14, 2001, which is a continuation-in-part of U.S. applicationSer. No. 09/290,508, entitled “Distributed Voice User Interface,” filedApr. 12, 1999, now U.S. Pat. No. 6,408,272, issued Jun. 18, 2002, all ofwhich are herein incorporated by reference in their entireties.

TECHNICAL FIELD OF THE INVENTION

The present invention relates generally to user interfaces and, moreparticularly, to a voice integration system.

BACKGROUND OF THE INVENTION

A voice user interface (VUI) allows a human user to interact with anintelligent, electronic device (e.g., a computer) by merely “talking” tothe device. The electronic device is thus able to receive, and respondto, directions, commands, instructions, or requests issued verbally bythe human user. As such, a VUI facilitates the use of the device.

A typical VUI is implemented using various techniques which enable anelectronic device to “understand” particular words or phrases spoken bythe human user, and to output or “speak” the same or differentwords/phrases for prompting, or responding to, the user. The words orphrases understood and/or spoken by a device constitute its“vocabulary.”

BRIEF SUMMARY OF THE INVENTION

The present invention provides a system and method for a voiceintegration system in which a voice gateway, service layers, and toolsinteract with an existing data system.

According to an embodiment of the present invention a voice integrationplatform provides for integration with a data system that includesstored data. The voice integration platform comprises one or moregeneric software components, the generic software components beingconfigured to enable development of a specific voice user interface thatis designed to interact with the data system in order to present thestored data to a user.

According to another embodiment of the present invention, a method isprovided. The method is for enabling the development of a voice userinterface that is designed to interact with a data system, where thedata system includes stored data. The method comprises providing one ormore generic software components, the generic software components beingconfigured to enable development of a specific voice user interface, thespecific voice user interface being designed to interact with the datasystem to present the stored data to a user.

According to yet another embodiment of the present invention, a voiceintegration platform provides for integration with a data system thatincludes stored data. The voice integration platform comprises means fordeveloping a specific voice user interface, the specific voice userinterface being designed to interact with the data system to present thestored data to a user.

Other aspects and advantages of the present invention will becomeapparent from the following descriptions and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS/FIGURES

For a more complete understanding of the present invention and forfurther features and advantages, reference is now made to the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 illustrates a voice integration platform according to at leastone embodiment of the present invention.

FIG. 2 illustrates a distributed voice user interface system, accordingto an embodiment of the present invention.

FIG. 3 illustrates details for a local device, according to anembodiment of the present invention.

FIG. 4 illustrates details for a remote system, according to anembodiment of the present invention.

FIG. 5 is a flow diagram of an exemplary method of operation for a localdevice, according to an embodiment of the present invention.

FIG. 6 is a flow diagram of an exemplary method of operation for aremote system, according to an embodiment of the present invention.

Like numerals are used for like and corresponding parts of the variousdrawings.

DETAILED DESCRIPTION OF THE INVENTION

Turning first to the nomenclature of the specification, the detaileddescription which follows is represented largely in terms of processesand symbolic representations of operations performed by conventionalcomputer components, such as a central processing unit (CPU) orprocessor associated with a general purpose computer system, memorystorage devices for the processor, and connected pixel-oriented displaydevices. These operations include the manipulation of data bits by theprocessor and the maintenance of these bits within data structuresresident in one or more of the memory storage devices. Such datastructures impose a physical organization upon the collection of databits stored within computer memory and represent specific electrical ormagnetic elements. These symbolic representations are the means used bythose skilled in the art of computer programming and computerconstruction to most effectively convey teachings and discoveries toothers skilled in the art.

For purposes of this discussion, a process, method, routine, orsub-routine is generally considered to be a sequence ofcomputer-executed steps leading to a desired result. These stepsgenerally require manipulations of physical quantities. Usually,although not necessarily, these quantities take the form of electrical,magnetic, or optical signals capable of being stored, transferred,combined, compared, or otherwise manipulated. It is conventional forthose skilled in the art to refer to these signals as bits, values,elements, symbols, characters, text, terms, numbers, records, objects,files, or the like. It should be kept in mind, however, that these andsome other terms should be associated with appropriate physicalquantities for computer operations, and that these terms are merelyconventional labels applied to physical quantities that exist within andduring operation of the computer.

It should also be understood that manipulations within the computer areoften referred to in terms such as adding, comparing, moving, or thelike, which are often associated with manual operations performed by ahuman operator. It must be understood that no involvement of the humanoperator may be necessary, or even desirable, in the present invention.The operations described herein are machine operations performed inconjunction with the human operator or user that interacts with thecomputer or computers.

In addition, it should be understood that the programs, processes,methods, and the like, described herein are but an exemplaryimplementation of the present invention and are not related, or limited,to any particular computer, apparatus, or computer language. Rather,various types of general purpose computing machines or devices may beused with programs constructed in accordance with the teachingsdescribed herein. Similarly, it may prove advantageous to construct aspecialized apparatus to perform the method steps described herein byway of dedicated computer systems with hard-wired logic or programsstored in non-volatile memory, such as read-only memory (ROM).

Voice Integration Platform Overview

Referring now to the drawings, FIG. 1 illustrates a voice integrationplatform 2, according to at least one embodiment of the presentinvention. In general, voice integration platform 2 provides voiceapplication design software.

When installed on a computer system that includes at least a processorand a memory, the voice integration platform 2 software provides a meansfor developing a specific voice user interface that is designed tointeract with an data system 6.

The voice integration design software of the voice integration platformincludes generic software components. These components are reusable,allowing the designer of a voice interface to utilize pre-written codein developing a specific voice application that is designed to interfacewith the data system 6 to deliver information (i.e., “stored data”) to auser. Using the voice integration platform 2, an interface designer, orteam of designers, can develop a voice application, such as a specificvoice user interface 27, that allows one or more human users 29 tointeract—via speech or verbal communication—with one or more datasystems 6. That is, data stored on the data system 6 is ordinarily notpresented to a user 29 via voice interaction. If the data system 6 is aWeb application server, for example, the user 29 requests and receivesinformation via a local device 14 (FIG. 2) using a standard GUIinterface to interact with the Web server. The voice integrationplatform 2 provides a development platform that enables an applicationdesigner to create a voice user interface 27 that interacts with storeddata on an existing data system 6. As used herein, the terms“connected,” “coupled,” or any variant thereof, means any connection orcoupling, either direct or indirect, between two or more elements; thecoupling or connection can be physical or logical.

FIG. 1 illustrates that the voice integration platform 2 is used togenerate a specific voice user interface 27. The specific voice userinterface 27 is designed to send data to, and receive data from, a datasystem 6. The data system 6 is any computer system having a memory thatstores data. In at least one embodiment, the data system 6 is a Web sitesystem that includes server hardware, software, and stored data suchthat the stored data can be delivered to a user via a network such asthe Internet. In at least one embodiment, the data system 6 is system ofone or more hardware servers, and software associated with an internetWeb site. In such embodiment, data system 6 typically contains one ormore Web application servers that include the necessary hardware andsoftware to store and serve HTML documents, associated files, andscripts to one or more local devices 14 (FIG. 2) when requested. The Webapplication servers are typically an Intel Pentium-based or RISC basedcomputer systems equipped with one or more processors, memory,input/output interfaces, a network interface, secondary storage devices,and a user interface. The stored data includes “pages” written inhypertext markup language (HTML) and may also include attribute andhistorical data concerning a specific user.

In at least one other embodiment, the data system 6 is a system thatsupports a customer call center application. Such system includes amemory that stores data associated with one or more customers, andsoftware and hardware that permit access to such stored data. In atraditional customer call center, the stored customer data ismanipulated, edited, and retrieved by human operators at computerterminals that have computerized access to the data. In such case, thedata may be delivered to the human operators via a mechanism other thanthe Internet, such as an internal network system. In at least one otherembodiment, the data system 6 is an automated banking system. Theforegoing specific examples of a data system 6 are for informationalpurposes only, and should not be taken to be limiting.

FIG. 1 illustrates that the voice integration platform 2 includes avoice gateway 4. The voice gateway 4, in at least one embodiment,incorporates at least some of the functionality of a distributed voiceuser interface (FIG. 2) described below. The voice gateway 4 allows theuser of a local device 14 (FIG. 2) to interact with the device 14 bytalking to the device 14.

FIG. 1 illustrates that the voice gateway is designed to work inconjunction with a set of service layers 3, 5, 7, 9, 11, 13. As usedherein, the term “service layer” refers to a set of one or more softwarecomponents that are logically grouped together based on sharedattributes. The service layers that interact with the voice gateway 4include the Applications service layer 3, the Personalized Dialogsservice layer 5, and the Infrastructure service layer 7. The voiceintegration platform 2 also includes a Personalization service layer 9,a Content Management service layer 11, and an Integration layer 13. Thelatter three service layers 9, 11, 13 is each capable of facilitatinginteraction between the data system 6 and any of the remaining threeservice layers 3, 5, and 7. A tools service layer 8 is designed to workin conjunction with the voice gateway 4 and each of the service layers3, 5, 7, 9, 11, 13. The tools service layer 8 set is a set of softwareprogramming tools that allows a voice application developer, forinstance, to monitor, test and debug voice application software codethat he develops using the voice integration platform 2.

Using the components of the voice integration platform 2 as adevelopment platform, a voice application designer can develop aspecific voice user interface 27 that is designed to integrate with aspecific existing data system 6. In at least one embodiment, theexisting data system 6 is the set of hardware, software, and data thatconstitute a Web site. Many other types of data systems arecontemplated, including automated banking systems, customer service callcenters, and the like.

Applications Service Layer

The applications service layer 3 includes components that add certainfunctional capabilities to the voice interface developed using the voiceintegration platform 2. For instance one of the components of theApplications service layer 3 is an email component 23. The emailcomponent 23 contains software, such as text-to-speech, speech-to-text,and directory management software, that provides the user the ability toreceive and send email messages in voice format. Another component ofthe Applications service layer 3 is a notification component 25. Thenotification component 25 provides for handing off information from thevoice user interface 27 to the local device 14 (FIG. 2) of a liveoperator when a user opts to transfer from an automated voiceapplication to live support.

Personalized Dialogs Layer

The Personalized Dialogs service layer 5 is a group of one or moresoftware components that allow a voice applications developer toincorporate natural language concepts into his product in order topresent a more human-like and conversational specific voice userinterface. The software components of the Personalized Dialogs servicelayer 5 implement rules for presenting voice information to a user inorder to emulate human dialog. Each of the software components mayinclude various constituents necessary for dialog emulation, such asVoiceXML scripts, .WAV files and audio files that make up the dialogpresented to the user, recognition grammars that are loaded into speechrecognition components, and software code for manipulating theconstituents as needed. For example, the Personalized Dialogs servicelayer 5 includes an error-trapping component 17. The error trappingcomponent 17 is a random prompt pool component. A specific example ofthis is the error-handling functionality, which has software logic thatprovides that prompts are not repeated when an error occurs with uservoice input. The error trapping component 17 includes code that mightprovide, upon an error condition, a prompt to the user that says, “Ididn't quite get that.” If the error condition is not corrected, insteadof repeating the prompt, the error trapping component might then providea prompt to the user that says, “Could you please repeat yourselection?” If the error condition is still not corrected, the errortrapping component 17 might then provide a prompt that says, “Well, I'mreally not understanding you.” By providing a series of distincterror-handling prompts rather than repeating the same prompt, a moreconversational dialog is carried on with the user than is provided byother voice interface systems.

As another example, the Personalized Dialogs service layer 5 includes alist browse component 19. The list browse component 19 provides forpresentation of a list of items to a user. The list browser componentimplements certain rules when presenting a list of information to a usersuch that the presentation emulates human verbal discourse.

Using the components of the Personalized Dialogs service layer 5, anapplication designer can design a voice user interface 27 that presentsdata to the user from an existing data system 6, presenting theinformation in a verbal format that is personalized to the particularuser. For instance, the voice user interface 27 can be designed toobtain attribute information about the user. This information could comedirectly from the user, in response to prompts, or from another sourcesuch as a cookie stored on the user's local device 14 (FIG. 2). Thevoice user interface 27 can also be designed to track historicalinformation among multiple sessions with a user, and even to trackhistorical information during a single user session. Using thisattribute and historical data, the components of the PersonalizedDialogs service layer 5 provide for personalized interaction with theuser. For an example that uses attribute data, the voice user interfaceprogrammed by the application designer (using the voice integrationplatform) speaks the user's name when interacting with the user.Similarly, if the user attribute data shows that the user lives in acertain U.S. city, the voice user interface can deliver local weatherinformation to the user. For an example using historical data acrossmore than one session, consider a voice user interface between a userand a data system 6 that provides banking services and data. If thevoice user interface 27 tracks historical information that indicatesthat a user, for 10 out of 11 previous sessions (whether conducting thesession using a voice interface or another interface such as a GUI),requested a checking account balance upon initiating the session, thenthe Personalized Dialogs service layer 5 provides for offering thechecking account balance to the user at the beginning of the session,without requiring that the user first request the data.

The Personalized Dialogs service layer 5 also provides for trackingother historical data and using that data to personalize dialogs withthe user. For instance, the service layer 5 can be utilized by theapplication programmer to provide for tracking user preference dataregarding advertisements presented to the user during a session. Forinstance, in at least one embodiment the voice integration platform 2provides for presenting voice advertisements to the user. ThePersonalized Dialogs service layer 2 keeps track of user actionregarding the advertisements. For instance, a voice ad might say, “GoodMorning, Joe, welcome to Global Bank's online service voice system.Would you like to hear about our new money market checking account?” ThePersonalized Dialogs service layer 5 provides a component that ensuresthat the format of the ad is rotated so that the wording is differentduring different sessions. For instance, during a different session thead might say, “Have you heard about our new money market checkingaccount?” The Personalized Dialog service layer contains a componentthat provides for tracking how many times a user has heard theadvertisement and tracks the user's historical responses to theadvertisement. To track the effectiveness of the ad, the PersonalizedDialogs service layer 5 keeps track of how many users opt to hear moreinformation about the advertised feature. By tracking user responses tovarious ads, user preference information is obtained. This historicaluser preference information is forwarded to the data system 6. Likewise,the Personalized Dialogs service layer 5 has access to historical andattribute data concerning a user that has been stored on the data system6. This data may come from any of several points of interaction, or“touchpoints”, between the user and the data system 6, includingtelephone access to a staffed call center, voice or non-voiceinteraction with the data system 6 from a local device such as apersonal computer or wireless device, and voice or non-voice telephonecommunications. This historical user preference information is alsomaintained for use by the Personalized Dialogs service layer 5. Thehistorical user preference information, along with preferenceinformation from the data system 6 that has been obtained during theuser's non-voice interaction with the data system 6, is used to providepersonalized dialogs to the user and to target specificpreference-responsive information to the user.

The Personalized Dialogs service layer 5 also includes a schedulingcomponent 21 that provides for scenario-driven personalization.Scenario-driven personalization provides additional interaction with theuser even after a voice session has been completed, depending on thetypes of actions taken by the user during the session. For instance, thescheduling component 21 provides an automated process for forwardingprinted material to a user if requested by the user during a session. Inaddition, in certain specified situations the scheduling component 21provides a notification (i.e., to a customer representative in a callcenter) to perform a follow-up call within a specified time period afterthe initial voice session.

Infrastructure Service Layer

The Infrastructure service layer 7 is a group of one or more softwarecomponents that are necessary for all specific voice user interfaces 27developed using the voice integration platform 2. For instance, theInfrastructure service layer 7 includes a domain controller softwarecomponent 15. The domain controller software component 15, alsosometimes referred to as a dialog manager, manages and controls theorganization and storage of information into logically distinct storagecategories referred to herein as “domains.” For instance, “electronicmail,” “sports scores,” “news,” and “stock quotes” are examples of fourdifferent domains. The domain controller software component 15 providesfor storage and retrieval of voice data in the appropriate domain. Insome instances, a piece of voice data may be relevant to more than onedomain. Accordingly, the domain controller software component 15provides for storage of the voice data in each of the appropriatedomains. The domain controller also traverses the stored domain data toretrieve user-specified data of interest.

Personalization Service Layer

The personalization service layer 9 contains software modules thatfacilitate interaction of the specific voice user interface 27 developedusing the voice integration platform 2 with personalization data in thedata system 6. For instance, the data system 6 may include code for apersonalization rules engine. The personalization rules engine on thedata system 6 can also be referred to as an inferencing engine. Theinferencing engine is software that accesses and processes data storedon the data system 6. For example, the inferencing engine in a datasystem that conducts e-commerce may track the types of purchases that aparticular user has made over time. Based on this information, theinferencing engine predicts other products or services that might be ofinterest to the particular user. In this manner, the data system 6generates a “recommended items” list for a particular user. ThePersonalization service layer 9 provides a software module thatfacilitates presentation of the “recommended items” to the user in voiceformat.

Content Management Service Layer

The Content Management service layer 11 contains one or more softwaremodules that facilitate interaction of the specific voice user interface27 developed using the voice integration platform 2 with contentmanagement software on the data system 6. For instance, a data system 6that manages a large amount of data may include content managementsoftware that classifies each file of data by associating a meta tagdescriptor with the file. This meta tag descriptor helps classify andidentify the contents of the data file. The Content Management servicelayer 11 provides a software module that facilitates access by thespecific voice user interface 27 developed using the voice integrationplatform 2 to the content management functionality, including meta tagdata, of the data system 6.

The Content Management service layer 11 also contains one or moresoftware components that provide for enhanced management of audiocontent. For instance, some audio files are streamed from a service tothe data system in broad categories. An example of this is the streamingof news and sports headlines to the data system 6 from the IndependentTelevision News (“ITN”) network. A content management software componentparses the stream of audio content to define constituent portions of thestream. The content management software module then associates eachdefined constituent portion with a particular domain. For instance, asports feed can be parsed into college sports and professional sportsitems that are then associated with the appropriate domain. For smallergranularity, the college sports items are further parsed and associatedwith football, baseball, basketball, and soccer domains. In this mannerthe content management software component provides smaller granularityon content than is provided as a streamed audio feed. One skilled in theart will understand that various types of audio data can be received bya data system 6, including voicemail, weather information, stock quotes,and email messages that have been converted to speech. Therefore, theexample concerning sports and news headlines audio feed should not betaken to be limiting.

In at least one embodiment, a content management software componentfacilitates generation of meta tag data for information received from anaudio feed, such as the ITN feed described above. The software componentprovides for converting the parsed audio files to text. Then, the textfiles are associated with meta data via interaction with the contentmanagement software on the data system 6.

In at least one embodiment, a content management software componentprovides templates for the creation of dialogs in a specific voice userinterface 27. This feature speeds the creation of dialogs and provides apre-tested environment for dialog creation that ensures that relatedcomponents, such as recognition grammars and audio files, are integratedproperly.

Integration Service Layer

The Integration service layer 13 is an input/output layer that containssoftware components for allowing the specific voice user interface 27and the data system 6 to exchange and share data.

Voice Gateway (Distributed VUI System)

FIG. 2 illustrates a distributed VUI system 10. Voice gateway 4 (FIG. 1)is a distributed VUI system 10. Distributed VUI system 10 includes aremote system 12 which may communicate with a number of local devices 14(separately designated with reference numerals 14 a, 14 b, 14 c, 14 d,14 e, 14 f, 14 g, 14 h, and 14 i) to implement one or more distributedVUIs. In one embodiment, a “distributed VUI” comprises a voice userinterface that may control the functioning of a respective local device14 through the services and capabilities of remote system 12. That is,remote system 12 cooperates with each local device 14 to deliver aseparate, sophisticated VUI capable of responding to a user andcontrolling that local device 14. In this way, the sophisticated VUIsprovided at local devices 14 by distributed VUI system 10 facilitate theuse of the local devices 14. In another embodiment, the distributed VUIenables control of another apparatus or system (e.g., a database or awebsite), in which case, the local device 14 serves as a “medium.”

Each such VUI of system 10 may be “distributed” in the sense that speechrecognition and speech output software and/or hardware can beimplemented in remote system 12 and the corresponding functionalitydistributed to the respective local device 14. Some speechrecognition/output software or hardware can be implemented in each oflocal devices 14 as well.

When implementing distributed VUI system 10 described herein, a numberof factors may be considered in dividing the speech recognition/outputfunctionality between local devices 14 and remote system 12. Thesefactors may include, for example, the amount of processing and memorycapability available at each of local devices 14 and remote system 12;the bandwidth of the link between each local device 14 and remote system12; the kinds of commands, instructions, directions, or requestsexpected from a user, and the respective, expected frequency of each;the expected amount of use of a local device 14 by a given user; thedesired cost for implementing each local device 14; etc. In oneembodiment, each local device 14 may be customized to address thespecific needs of a particular user, thus providing a technicaladvantage.

Local Devices

Each local device 14 can be an electronic device with a processor havinga limited amount of processing or computing power. For example, a localdevice 14 can be a relatively small, portable, inexpensive, and/or lowpower-consuming “smart device,” such as a personal digital assistant(PDA), a wireless remote control (e.g., for a television set or stereosystem), a smart telephone (such as a cellular phone or a stationaryphone with a screen), or smart jewelry (e.g., an electronic watch). Alocal device 14 may also comprise or be incorporated into a largerdevice or system, such as a television set, a television set top box(e.g., a cable receiver, a satellite receiver, or a video game station),a video cassette recorder, a video disc player, a radio, a stereosystem, an automobile dashboard component, a microwave oven, arefrigerator, a household security system, a climate control system (forheating and cooling), or the like.

In one embodiment, a local device 14 uses elementary techniques (e.g.,the push of a button) to detect the onset of speech. Local device 14then performs preliminary processing on the speech waveform. Forexample, local device 14 may transform speech into a series of featurevectors or frequency domain parameters (which differ from the digitizedor compressed speech used in vocoders or cellular phones). Specifically,from the speech waveform, the local device 14 may extract variousfeature parameters, such as, for example, cepstral coefficients, Fouriercoefficients, linear predictive coding (LPC) coefficients, or otherspectral parameters in the time or frequency domain. These spectralparameters (also referred to as features in automatic speech recognitionsystems), which would normally be extracted in the first stage of aspeech recognition system, are transmitted to remote system 12 forprocessing therein. Speech recognition and/or speech outputhardware/software at remote system 12 (in communication with the localdevice 14) then provides a sophisticated VUI through which a user caninput commands, instructions, or directions into, and/or retrieveinformation or obtain responses from, the local device 14.

In another embodiment, in addition to performing preliminary signalprocessing (including feature parameter extraction), at least a portionof local devices 14 may each be provided with its own resident VUI. Thisresident VUI allows the respective local device 14 to understand andspeak to a user, at least on an elementary level, without remote system12. To accomplish this, each such resident VUI may include, or becoupled to, suitable input/output devices (e.g., microphone and speaker)for receiving and outputting audible speech. Furthermore, each residentVUI may include hardware and/or software for implementing speechrecognition (e.g., automatic speech recognition (ASR) software) andspeech output (e.g., recorded or generated speech output software). Anexemplary embodiment for a resident VUI of a local device 14 isdescribed below in more detail.

A local device 14 with a resident VUI may be, for example, a remotecontrol for a television set. A user may issue a command to the localdevice 14 by stating “Channel four” or “Volume up,” to which the localdevice 14 responds by changing the channel on the television set tochannel four or by turning up the volume on the set.

Because each local device 14, by definition, has a processor withlimited computing power, the respective resident VUI for a local device14, taken alone, generally does not provide extensive speech recognitionand/or speech output capability. For example, rather than implement amore complex and sophisticated natural language (NL) technique forspeech recognition, each resident VUI may perform “word spotting” byscanning speech input for the occurrence of one or more “keywords.”Furthermore, each local device 14 will have a relatively limitedvocabulary (e.g., less than one hundred words) for its resident VUI. Assuch, a local device 14, by itself, is only capable of responding torelatively simple commands, instructions, directions, or requests from auser.

In instances where the speech recognition and/or speech outputcapability provided by a resident VUI of a local device 14 is notadequate to address the needs of a user, the resident VUI can besupplemented with the more extensive capability provided by remotesystem 12. Thus, the local device 14 can be controlled by spokencommands and otherwise actively participate in verbal exchanges with theuser by utilizing more complex speech recognition/output hardware and/orsoftware implemented at remote system 12 (as further described herein).

Each local device 14 may further comprise a manual input device—such asa button, a toggle switch, a keypad, or the like—by which a user caninteract with the local device 14 (and also remote system 12 via asuitable communication network) to input commands, instructions,requests, or directions without using either the resident or distributedVUI. For example, each local device 14 may include hardware and/orsoftware supporting the interpretation and issuance of dual tonemultiple frequency (DTMF) commands. In one embodiment, such manual inputdevice can be used by the user to activate or turn on the respectivelocal device 14 and/or initiate communication with remote system 12.

Remote System

In general, remote system 12 supports a relatively sophisticated VUIwhich can be utilized when the capabilities of any given local device 14alone are insufficient to address or respond to instructions, commands,directions, or requests issued by a user at the local device 14. The VUIat remote system 12 can be implemented with speech recognition/outputhardware and/or software suitable for performing the functionalitydescribed herein.

The VUI of remote system 12 interprets the vocalized expressions of auser—communicated from a local device 14—so that remote system 12 mayitself respond, or alternatively, direct the local device 14 to respond,to the commands, directions, instructions, requests, and other inputspoken by the user. As such, remote system 12 completes the task ofrecognizing words and phrases.

The VUI at remote system 12 can be implemented with a different type ofautomatic speech recognition (ASR) hardware/software than local devices14. For example, in one embodiment, rather than performing “wordspotting,” as may occur at local devices 14, remote system 12 may use alarger vocabulary recognizer, implemented with word and optionalsentence recognition grammars. A recognition grammar specifies a set ofdirections, commands, instructions, or requests that, when spoken by auser, can be understood by a VUI. In other words, a recognition grammarspecifies what sentences and phrases are to be recognized by the VUI.For example, if a local device 14 comprises a microwave oven, adistributed VUI for the same can include a recognition grammar thatallows a user to set a cooking time by saying, “Oven high for half aminute,” or “Cook on high for thirty seconds,” or, alternatively,“Please cook for thirty seconds at high.” Commercially available speechrecognition systems with recognition grammars are provided by ASRtechnology vendors such as, for example, the following: NuanceCorporation of Menlo Park, Calif.; Dragon Systems of Newton, Mass.; IBMof Austin, Tex.; Kurzweil Applied Intelligence of Waltham, Mass.;Lernout Hauspie Speech Products of Burlington, Mass.; and PureSpeech,Inc. of Cambridge, Mass.

Remote system 12 may process the directions, commands, instructions, orrequests that it has recognized or understood from the utterances of auser. During processing, remote system 12 can, among other things,generate control signals and reply messages, which are returned to alocal device 14. Control signals are used to direct or control the localdevice 14 in response to user input. For example, in response to a usercommand of “Turn up the heat to 82 degrees,” control signals may directa local device 14 incorporating a thermostat to adjust the temperatureof a climate control system. Reply messages are intended for theimmediate consumption of a user at the local device and may take theform of video or audio, or text to be displayed at the local device. Asa reply message, the VUI at remote system 12 may issue audible output inthe form of speech that is understandable by a user.

For issuing reply messages, the VUI of remote system 12 may includecapability for speech generation (synthesized speech) and/or play-back(previously recorded speech). Speech generation capability can beimplemented with text-to-speech (TTS) hardware/software, which convertstextual information into synthesized, audible speech. Speech play-backcapability may be implemented with an analog-to-digital (A/D) converterdriven by CD ROM (or other digital memory device), a tape player, alaser disc player, a specialized integrated circuit (IC) device, or thelike, which plays back previously recorded human speech.

In speech play-back, a person (preferably a voice model) recites variousstatements which may desirably be issued during an interactive sessionwith a user at a local device 14 of distributed VUI system 10. Theperson's voice is recorded as the recitations are made. The recordingsare separated into discrete messages, each message comprising one ormore statements that would desirably be issued in a particular context(e.g., greeting, farewell, requesting instructions, receivinginstructions, etc.). Afterwards, when a user interacts with distributedVUI system 10, the recorded messages are played back to the user whenthe proper context arises.

The reply messages generated by the VUI at remote system 12 can be madeto be consistent with any messages provided by the resident VUI of alocal device 14. For example, if speech play-back capability is used forgenerating speech, the same person's voice may be recorded for messagesoutput by the resident VUI of the local device 14 and the VUI of remotesystem 12. If synthesized (computer-generated) speech capability isused, a similar sounding artificial voice may be provided for the VUIsof both local devices 14 and remote system 12. In this way, thedistributed VUI of system 10 provides to a user an interactive interfacewhich is “seamless” in the sense that the user cannot distinguishbetween the simpler, resident VUI of the local device 14 and the moresophisticated VUI of remote system 12.

In one embodiment, the speech recognition and speech play-backcapabilities described herein can be used to implement a voice userinterface with personality, as taught by U.S. patent application Ser.No. 09/071,717, entitled “Voice User Interface With Personality,” thetext of which is incorporated herein by reference.

Remote system 12 may also comprise hardware and/or software supportingthe interpretation and issuance of commands, such as dual tone multiplefrequency (DTMF) commands, so that a user may alternatively interactwith remote system 12 using an alternative input device, such as atelephone key pad.

Remote system 12 may be in communication with the “Internet,” thusproviding access thereto for users at local devices 14. The Internet isan interconnection of computer “clients” and “servers” locatedthroughout the world and exchanging information according toTransmission Control Protocol/Internet Protocol (TCP/IP), InternetworkPacket eXchange/Sequence Packet eXchange (IPX/SPX), AppleTalk, or othersuitable protocol. The Internet supports the distributed applicationknown as the “World Wide Web.” Web servers may exchange information withone another using a protocol known as hypertext transport protocol(HTTP). Information may be communicated from one server to any othercomputer using HTTP and is maintained in the form of web pages, each ofwhich can be identified by a respective uniform resource locator (URL).Remote system 12 may function as a client to interconnect with Webservers. The interconnection may use any of a variety of communicationlinks, such as, for example, a local telephone communication line or adedicated communication line. Remote system 12 may comprise and locallyexecute a “web browser” or “web proxy” program. A web browser is acomputer program that allows remote system 12, acting as a client, toexchange information with the World Wide Web. Any of a variety of webbrowsers are available, such as NETSCAPE NAVIGATOR from AOL Time WarnerInc. of New York, N.Y., INTERNET EXPLORER from Microsoft Corporation ofRedmond, Wash., and others that allow users to conveniently access andnavigate the Internet. A web proxy is a computer program which (via theInternet) can, for example, electronically integrate the systems of acompany and its vendors and/or customers, support business transactedelectronically over the network (i.e., “e-commerce”), and provideautomated access to Web-enabled resources. Any number of web proxies areavailable, such as B2B INTEGRATION SERVER from webMethods of Fairfax,Va., and MICROSOFT PROXY SERVER from Microsoft Corporation of Redmond,Wash. The hardware, software, and protocols—as well as the underlyingconcepts and techniques—supporting the Internet are generally understoodby those in the art.

Communication Network

One or more suitable communication networks enable local devices 14 tocommunicate with remote system 12. For example, as shown, local devices14 a, 14 b, and 14 c communicate with remote system 12 viatelecommunications network 16; local devices 14 d, 14 e, and 14 fcommunicate via local area network (LAN) 18; and local devices 14 g, 14h, and 14 i communicate via the Internet.

Telecommunications network 16 allows a user to interact with remotesystem 12 from a local device 14 via a telecommunications line, such asan analog telephone line, a digital T1 line, a digital T3 line, or anOC3 telephony feed. Telecommunications network 16 may include a publicswitched telephone network (PSTN) and/or a private system (e.g.,cellular system) implemented with a number of switches, wire lines,fiber-optic cable, land-based transmission towers, space-based satellitetransponders, etc. In one embodiment, telecommunications network 16 mayinclude any other suitable communication system, such as a specializedmobile radio (SMR) system. As such, telecommunications network 16 maysupport a variety of communications, including, but not limited to,local telephony, toll (i.e., long distance), and wireless (e.g., analogcellular system, digital cellular system, Personal Communication System(PCS), Cellular Digital Packet Data (CDPD), ARDIS, RAM Mobile Data,Metricom Ricochet, paging, and Enhanced Specialized Mobile Radio(ESMR)). Telecommunications network 16 may utilize various callingprotocols (e.g., Inband, Integrated Services Digital Network (ISDN) andSignaling System No. 7 (SS7) call protocols) and other suitableprotocols (e.g., Enhanced Throughput Cellular (ETC), Enhanced CellularControl (EC2), MNP 10, MNP10-EC, Throughput Accelerator (TXCEL), MobileData Link Protocol, etc.). Transmissions over telecommunications networksystem 16 may be analog or digital. Transmission may also include one ormore infrared links (e.g., IRDA).

In general, local area network (LAN) 18 connects a number of hardwaredevices in one or more of various configurations or topologies, whichmay include, for example, Ethernet, token ring, and star, and provides apath (e.g., bus) which allows the devices to communicate with eachother. With local area network 18, multiple users are given access to acentral resource. As depicted, users at local devices 14 d, 14 e, and 14f are given access to remote system 12 for provision of the distributedVUI.

For communication over the Internet, remote system 12 and/or localdevices 14 g, 14 h, and 14 i may be connected to, or incorporate,servers and clients communicating with each other using the protocols(e.g., TCP/IP or UDP), addresses (e.g., URL), links (e.g., dedicatedline), and browsers (e.g., NETSCAPE NAVIGATOR) described above.

As an alternative, or in addition, to telecommunications network 16,local area network 18, or the Internet (as depicted in FIG. 1),distributed VUI system 10 may utilize one or more other suitablecommunication networks. Such other communication networks may compriseany suitable technologies for transmitting/receiving analog or digitalsignals. For example, such communication networks may comprise cablemodems, satellite, radio, and/or infrared links.

The connection provided by any suitable communication network (e.g.,telecommunications network 16, local area network 18, or the Internet)can be transient. That is, the communication network need notcontinuously support communication between local devices 14 and remotesystem 12, but rather, only provides data and signal transfer therebetween when a local device 14 requires assistance from remote system12. Accordingly, operating costs (e.g., telephone facility charges) fordistributed VUI system 10 can be substantially reduced or minimized.

Operation of Voice Gateway (In General)

In generalized operation, each local device 14 can receive input in theform of vocalized expressions (i.e., speech input) from a user and mayperform preliminary or initial signal processing, such as, for example,feature extraction computations and elementary speech recognitioncomputations. The local device 14 then determines whether it is capableof further responding to the speech input from the user. If not, localdevice 14 communicates—for example, over a suitable network, such astelecommunications network 16 or local area network (LAN) 18—with remotesystem 12. Remote system 12 performs its own processing, which mayinclude more advanced speech recognition techniques and the accessing ofother resources (e.g., data available on the Internet). Afterwards,remote system 12 returns a response to the local device 14. Suchresponse can be in the form of one or more reply messages and/or controlsignals. The local device 14 delivers the messages to its user, and thecontrol signals modify the operation of the local device 14.

Local Device (Details)

FIG. 3 illustrates details for a local device 14, according to anembodiment of the present invention. As depicted, local device 14comprises a primary functionality component 19, a microphone 20, aspeaker 22, a manual input device 24, a display 26, a processingcomponent 28, a recording device 30, and a transceiver 32.

Primary functionality component 19 performs the primary functions forwhich the respective local device 14 is provided. For example, if localdevice 14 comprises a personal digital assistant (PDA), primaryfunctionality component 19 can maintain a personal organizer whichstores information for names, addresses, telephone numbers, importantdates, appointments, and the like. Similarly, if local device 14comprises a stereo system, primary functionality component 19 can outputaudible sounds for a user's enjoyment by tuning into radio stations,playing tapes or compact discs, etc. If local device 14 comprises amicrowave oven, primary functionality component 19 can cook foods.Primary functionality component 19 may be controlled by control signalswhich are generated by the remainder of local device 14, or remotesystem 12, in response to a user's commands, instructions, directions,or requests. Primary functionality component 19 is optional, andtherefore, may not be present in every implementation of a local device14; such a device could be one having a sole purpose of sending ortransmitting information.

Microphone 20 detects the audible expressions issued by a user andrelays the same to processing component 28 for processing within aparameter extraction component 34 and/or a resident voice user interface(VUI) 36 contained therein. Speaker 22 outputs audible messages orprompts which can originate from resident VUI 36 of local device 14, oralternatively, from the VUI at remote system 12. Speaker 22 is optional,and therefore, may not be present in every implementation; for example,a local device 14 can be implemented such that output to a user is viadisplay 26 or primary functionality component 19.

Manual input device 24 comprises a device by which a user can manuallyinput information into local device 14 for any of a variety of purposes.For example, manual input device 24 may comprise a keypad, button,switch, or the like, which a user can depress or move toactivate/deactivate local device 14, control local device 14, initiatecommunication with remote system 12, input data to remote system 12,etc. Manual input device 24 is optional, and therefore, may not bepresent in every implementation; for example, a local device 14 can beimplemented such that user input is via microphone 20 only. Display 26comprises a device, such as, for example, a liquid-crystal display (LCD)or light-emitting diode (LED) screen, which displays data visually to auser. In some embodiments, display 26 may comprise an interface toanother device, such as a television set. Display 26 is optional, andtherefore, may not be present in every implementation; for example, alocal device 14 can be implemented such that user output is via speaker22 only.

Processing component 28 is connected to each of primary functionalitycomponent 19, microphone 20, speaker 22, manual input device 24, anddisplay 26. In general, processing component 28 provides processing orcomputing capability in local device 14. In one embodiment, processingcomponent 28 may comprise a microprocessor connected to (orincorporating) supporting memory to provide the functionality describedherein. As previously discussed, such a processor has limited computingpower.

Processing component 28 may output control signals to primaryfunctionality component 19 for control thereof. Such control signals canbe generated in response to commands, instructions, directions, orrequests which are spoken by a user and interpreted or recognized byresident VUI 36 and/or remote system 12. For example, if local device 14comprises a household security system, processing component 28 mayoutput control signals for disarming the security system in response toa user's verbalized command of “Security off, code 4-2-5-6-7.”

Parameter extraction component 34 may perform a number of preliminarysignal processing operations on a speech waveform. Among other things,these operations transform speech into a series of feature parameters,such as standard cepstral coefficients, Fourier coefficients, linearpredictive coding (LPC) coefficients, or other parameters in thefrequency or time domain. For example, in one embodiment, parameterextraction component 34 may produce a twelve-dimensional vector ofcepstral coefficients every ten milliseconds to model speech input data.Software for implementing parameter extraction component 34 iscommercially available from line card manufacturers and ASR technologysuppliers such as Dialogic Corporation of Parsippany, N.J., and NaturalMicroSystems Inc. of Natick, Mass.

Resident VUI 36 may be implemented in processing component 28. Ingeneral, VUI 36 allows local device 14 to understand and speak to a useron at least an elementary level. As shown, VUI 36 of local device 14 mayinclude a barge-in component 38, a speech recognition engine 40, and aspeech generation engine 42.

Barge-in component 38 generally functions to detect speech from a userat microphone 20 and, in one embodiment, can distinguish human speechfrom ambient background noise. When speech is detected by barge-incomponent 38, processing component 28 ceases to emit any speech which itmay currently be outputting so that processing component 28 can attendto the new speech input. Thus, a user is given the impression that he orshe can interrupt the speech generated by local device 14 (and thedistributed VUI system 10) simply by talking. Software for implementingbarge-in component 38 is commercially available from line cardmanufacturers and ASR technology suppliers such as Dialogic Corporationof Parsippany, N.J., and Natural MicroSystems Inc. of Natick, Mass.Barge-in component 38 is optional, and therefore, may not be present inevery implementation.

Speech recognition engine 40 can recognize speech at an elementarylevel, for example, by performing keyword searching. For this purpose,speech recognition engine 40 may comprise a keyword search component 44which is able to identify and recognize a limited number (e.g., 100 orless) of keywords. Each keyword may be selected in advance based uponcommands, instructions, directions, or requests which are expected to beissued by a user. In one embodiment, speech recognition engine 40 maycomprise a logic state machine. Speech recognition engine 40 can beimplemented with automatic speech recognition (ASR) softwarecommercially available, for example, from the following companies:Nuance Corporation of Menlo Park, Calif.; Applied Language Technologies,Inc. of Boston, Mass.; Dragon Systems of Newton, Mass.; and PureSpeech,Inc. of Cambridge, Mass. Such commercially available software typicallycan be modified for particular applications, such as a computertelephony application. As such, the resident VUI 36 can be configured ormodified by a user or another party to include a customized keywordgrammar. In one embodiment, keywords for a grammar can be downloadedfrom remote system 12. In this way, keywords already existing in localdevice 14 can be replaced, supplemented, or updated as desired.

Speech generation engine 42 can output speech, for example, by playingback pre-recorded messages, to a user at appropriate times. For example,several recorded prompts and/or responses can be stored in the memory ofprocessing component 28 and played back at any appropriate time. Suchplay-back capability can be implemented with a play-back component 46comprising suitable hardware/software, which may include an integratedcircuit device. In one embodiment, pre-recorded messages (e.g., promptsand responses) may be downloaded from remote system 12. In this manner,the pre-recorded messages already existing in local device 14 can bereplaced, supplemented, or updated as desired. Speech generation engine42 is optional, and therefore, may not be present in everyimplementation; for example, a local device 14 can be implemented suchthat user output is via display 26 or primary functionality component 19only.

Recording device 30, which is connected to processing component 28,functions to maintain a record of each interactive session with a user(i.e., interaction between distributed VUI system 10 and a user afteractivation, as described below). Such record may include the verbalutterances issued by a user during a session and preliminarily processedby parameter extraction component 34 and/or resident VUI 36. Theserecorded utterances are exemplary of the language used by a user andalso the acoustic properties of the user's voice. The recordedutterances can be forwarded to remote system 12 for further processingand/or recognition. In a robust technique, the recorded utterances canbe analyzed (for example, at remote system 12) and the keywordsrecognizable by distributed VUI system 10 updated or modified accordingto the user's word choices. The record maintained at recording device 30may also specify details for the resources or components used inmaintaining, supporting, or processing the interactive session. Suchresources or components can include microphone 20, speaker 22,telecommunications network 16, local area network 18, connection charges(e.g., telecommunications charges), etc. Recording device 30 can beimplemented with any suitable hardware/software. Recording device 30 isoptional, and therefore, may not be present in some implementations.

Transceiver 32 is connected to processing component 28 and functions toprovide bi-directional communication with remote system 12 overtelecommunications network 16. Among other things, transceiver 32 maytransfer speech and other data to and from local device 14. Such datamay be coded, for example, using 32-KB Adaptive Differential Pulse CodedModulation (ADPCM) or 64-KB MU-law parameters using commerciallyavailable modulation devices from, for example, Rockwell Internationalof Newport Beach, Calif. In addition, or alternatively, speech data maybe transfer coded as LPC parameters or other parameters achieving lowbit rates (e.g., 4.8 Kbits/sec), or using a compressed format, such as,for example, with commercially available software from Voxware ofPrinceton, N.J. Data sent to remote system 12 can include frequencydomain parameters extracted from speech by processing component 28. Datareceived from remote system 12 can include that supporting audio and/orvideo output at local device 14, and also control signals forcontrolling primary functionality component 19. The connection fortransmitting data to remote system 12 can be the same or different fromthe connection for receiving data from remote system 12. In oneembodiment, a “high bandwidth” connection is used to return data forsupporting audio and/or video, whereas a “low bandwidth” connection maybe used to return control signals.

In one embodiment, in addition to, or in lieu of, transceiver 32, localdevice 14 may comprise a local area network (LAN) connector and/or awide area network (WAN) connector (neither of which are explicitlyshown) for communicating with remote system 12 via local area network 18or the Internet, respectively. The LAN connector can be implemented withany device which is suitable for the configuration or topology (e.g.,Ethernet, token ring, or star) of local area network 18. The WANconnector can be implemented with any device (e.g., router) supportingan applicable protocol (e.g., TCP/IP, IPX/SPX, or AppleTalk).

Local device 14 may be activated upon the occurrence of any one or moreactivation or triggering events. For example, local device 14 mayactivate at a predetermined time (e.g., 7:00 a.m. each day), at thelapse of a predetermined interval (e.g., twenty-four hours), or upontriggering by a user at manual input operating—listening to speechissued from a user, extracting feature parameters (e.g., cepstral,Fourier, or LPC) from the speech, and/or scanning for keyword “wake up”phrases.

After activation and during operation, when a user verbally issuescommands, instructions, directions, or requests at microphone 20 orinputs the same at manual input device 24, local device 14 may respondby outputting control signals to primary functionality component 19and/or outputting speech to the user at speaker 22. If local device 14is able, it generates these control signals and/or speech by itselfafter processing the user's commands, instructions, directions, orrequests, for example, within resident VUI 36. If local device 14 is notable to respond by itself (e.g., it cannot recognize a user's spokencommand) or, alternatively, if a user triggers local device 14 with a“wake up” command, local device 14 initiates communication with remotesystem 12. Remote system 12 may then process the spoken commands,instructions, directions, or requests at its own VUI and return controlsignals or speech to local device 14 for forwarding to primaryfunctionality component 19 or a user, respectively.

For example, local device 14 may, by itself, be able to recognize andrespond to an instruction of “Dial number 555-1212,” but may require theassistance of remote device 12 to respond to a request of “What is theweather like in Chicago?”

Remote System (Details)

FIG. 4 illustrates details for a remote system 12, according to anembodiment of the present invention. Remote system 12 may cooperate withlocal devices 14 to provide a distributed VUI for communication withrespective users and to generate control signals for controllingrespective primary functionality components 19. As depicted, remotesystem 12 comprises a transceiver 50, a LAN connector 52, a processingcomponent 54, a memory 56, and a WAN connector 58. Depending on thecombination of local devices 14 supported by remote system 12, only oneof the following may be required, with the other two optional:transceiver 50, LAN connector 52, or WAN connector 58.

Transceiver 50 provides bi-directional communication with one or morelocal devices 14 over telecommunications network 16. As shown,transceiver 50 may include a telephone line card 60 which allows remotesystem 12 to communicate with telephone lines, such as, for example,analog telephone lines, digital Ti lines, digital T3 lines, or OC3telephony feeds. Telephone line card 60 can be implemented with variouscommercially available telephone line cards from, for example, DialogicCorporation of Parsippany, N.J. (which supports twenty-four lines) orNatural MicroSystems Inc. of Natick, Mass. (which supports from two toforty-eight lines). Among other things, transceiver 50 may transferspeech data to and from local device 14. Speech data can be coded as,for example, 32-KB Adaptive Differential Pulse Coded Modulation (ADPCM)or 64-KB MU-law parameters using commercially available modulationdevices from, for example, Rockwell International of Newport Beach,Calif. In addition, or alternatively, speech data may be transfer codedas LPC parameters or other parameters achieving low bit rates (e.g., 4.8Kbits/sec), or using a compressed format, such as, for example, withcommercially available software from Voxware of Princeton, N.J.

LAN connector 52 allows remote system 12 to communicate with one or morelocal devices over local area network 18. LAN connector 52 can beimplemented with any device supporting the configuration or topology(e.g., Ethernet, token ring, or star) of local area network 18. LANconnector 52 can be implemented with a LAN card commercially availablefrom, for example, 3COM Corporation of Santa Clara, Calif.

Processing component 54 is connected to transceiver 50 and LAN connector52. In general, processing component 54 provides processing or computingcapability in remote system 12. The functionality of processingcomponent 54 can be performed by any suitable processor, such as amain-frame, a file server, a workstation, or other suitable dataprocessing facility supported by memory (either internal or external)and running appropriate software. In one embodiment, processingcomponent 54 can be implemented as a physically distributed orreplicated system. Processing component 54 may operate under the controlof any suitable operating system (OS), such as MS-DOS, MacINTOSH OS,WINDOWS NT, WINDOWS 95, OS/2, UNIX, LINUX, XENIX, and the like.

Processing component 54 may receive—from transceiver 50, LAN connector52, and WAN connector 58—commands, instructions, directions, orrequests, issued by one or more users at local devices 14. Processingcomponent 54 processes these user commands, instructions, directions, orrequests and, in response, may generate control signals or speechoutput.

For recognizing and outputting speech, a VUI 62 is implemented inprocessing component 54. This VUI 62 is more sophisticated than theresident VUIs 34 of local devices 14. For example, VUI 62 can have amore extensive vocabulary with respect to both the word/phrases whichare recognized and those which are output. VUI 62 of remote system 12can be made to be consistent with resident VUIs 34 of local devices 14.For example, the messages or prompts output by VUI 62 and VUIs 34 can begenerated in the same synthesized, artificial voice. Thus, VUI 62 andVUIs 34 operate to deliver a “seamless” interactive interface to a user.In some embodiments, multiple instances of VUI 62 may be provided suchthat a different VUI is used based on the type of local device 14. Asshown, VUI 62 of remote system 12 may include an echo cancellationcomponent 64, a barge-in component 66, a signal processing component 68,a speech recognition engine 70, and a speech generation engine 72.

Echo cancellation component 64 removes echoes caused by delays (e.g., intelecommunications network 16) or reflections from acoustic waves in theimmediate environment of a local device 14. This provides “higherquality” speech for recognition and processing by VUI 62. Software forimplementing echo cancellation component 64 is commercially availablefrom Noise Cancellation Technologies of Stamford, Conn.

Barge-in component 66 may detect speech received at transceiver 50, LANconnector 52, or WAN connector 58. In one embodiment, barge-in component66 may distinguish human speech from ambient background noise. Whenbarge-in component 66 detects speech, any speech output by thedistributed VUI is halted so that VUI 62 can attend to the new speechinput. Software for implementing barge-in component 66 is commerciallyavailable from line card manufacturers and ASR technology suppliers suchas, for example, Dialogic Corporation of Parsippany, N.J., and NaturalMicroSystems Inc. of Natick, Mass. Barge-in component 66 is optional,and therefore, may not be present in every implementation.

Signal processing component 68 performs signal processing operationswhich, among other things, may include transforming speech data receivedin time domain format (such as ADPCM) into a series of featureparameters such as, for example, standard cepstral coefficients, Fouriercoefficients, linear predictive coding (LPC) coefficients, or otherparameters in the time or frequency domain. For example, in oneembodiment, signal processing component 68 may produce atwelve-dimensional vector of cepstral coefficients every 10 millisecondsto model speech input data. Software for implementing signal processingcomponent 68 is commercially available from line card manufacturers andASR technology suppliers such as Dialogic Corporation of Parsippany,N.J., and Natural MicroSystems Inc. of Natick, Mass.

Speech recognition engine 70 allows remote system 12 to recognizevocalized speech. As shown, speech recognition engine 70 may comprise anacoustic model component 73 and a grammar component 74. Acoustic modelcomponent 73 may comprise one or more reference voice templates whichstore previous enunciations (or acoustic models) of certain words orphrases by particular users. Acoustic model component 73 recognizes thespeech of the same users based upon their previous enunciations storedin the reference voice templates. Grammar component 74 may specifycertain words, phrases, and/or sentences which are to be recognized ifspoken by a user. Recognition grammars for grammar component 74 can bedefined in a grammar definition language (GDL), and the recognitiongrammars specified in GDL can then be automatically translated intomachine executable grammars. In one embodiment, grammar component 74 mayalso perform natural language (NL) processing. Hardware and/or softwarefor implementing a recognition grammar is commercially available fromsuch vendors as the following: Nuance Corporation of Menlo Park, Calif.;Dragon Systems of Newton, Mass.; IBM of Austin, Tex.; Kurzweil AppliedIntelligence of Waltham, Mass.; Lernout Hauspie Speech Products ofBurlington, Mass.; and PureSpeech, Inc. of Cambridge, Mass. Naturallanguage processing techniques can be implemented with commercialsoftware products separately available from, for example, UNISYSCorporation of Blue Bell, Pa. These commercially availablehardware/software can typically be modified for particular applications.

Speech generation engine 72 allows remote system 12 to issue verbalizedresponses, prompts, or other messages, which are intended to be heard bya user at a local device 14. As depicted, speech generation engine 72comprises a text-to-speech (TTS) component 76 and a play-back component78. Text-to-speech component 76 synthesizes human speech by “speaking”text, such as that contained in a textual e-mail document.Text-to-speech component 76 may utilize one or more synthetic speechmark-up files for determining, or containing, the speech to besynthesized. Software for implementing text-to-speech component 76 iscommercially available, for example, from the following companies:AcuVoice, Inc. of San Jose, Calif.; Centigram Communications Corporationof San Jose, Calif.; Digital Equipment Corporation (DEC) of Maynard,Mass.; Lucent Technologies of Murray Hill, N.J.; and Entropic ResearchLaboratory, Inc. of Washington, D.C. Play-back component 78 plays backpre-recorded messages to a user. For example, several thousand recordedprompts or responses can be stored in memory 56 of remote system 12 andplayed back at any appropriate time. Speech generation engine 72 isoptional (including either or both of text-to-speech component 76 andplay-back component 78), and therefore, may not be present in everyimplementation.

Memory 56 is connected to processing component 54. Memory 56 maycomprise any suitable storage medium or media, such as random accessmemory (RAM), read-only memory (ROM), disk, tape storage, or othersuitable volatile and/or non-volatile data storage system. Memory 56 maycomprise a relational database. Memory 56 receives, stores, and forwardsinformation which is utilized within remote system 12 and, moregenerally, within distributed VUI system 10. For example, memory 56 maystore the software code and data supporting the acoustic models,grammars, text-to-speech, and play-back capabilities of speechrecognition engine 70 and speech generation engine 72 within VUI 64.

WAN connector 58 is coupled to processing component 54. WAN connector 58enables remote system 12 to communicate with the Internet using, forexample, Transmission Control Protocol/Internet Protocol (TCP/IP),Internetwork Packet eXchange/Sequence Packet eXchange (IPX/SPX),AppleTalk, or any other suitable protocol. By supporting communicationwith the Internet, WAN connector 58 allows remote system 12 to accessvarious remote databases containing a wealth of information (e.g., stockquotes, telephone listings, directions, news reports, weather and travelinformation, etc.) which can be retrieved/downloaded and ultimatelyrelayed to a user at a local device 14. WAN connector 58 can beimplemented with any suitable device or combination of devices—such as,for example, one or more routers and/or switches—operating inconjunction with suitable software. In one embodiment, WAN connector 58supports communication between remote system 12 and one or more localdevices 14 over the Internet.

Operation at Local Device

FIG. 5 is a flow diagram of an exemplary method 100 of operation for alocal device 14, according to an embodiment of the present invention.

Method 100 begins at step 102 where local device 14 waits for someactivation event, or particular speech issued from a user, whichinitiates an interactive user session, thereby activating processingwithin local device 14. Such activation event may comprise the lapse ofa predetermined interval (e.g., twenty-four hours) or triggering by auser at manual input device 24, or may coincide with a predeterminedtime (e.g., 7:00 a.m. each day). In another embodiment, the activationevent can be speech from a user. Such speech may comprise one or morecommands in the form of keywords—e.g., “Start,” “Turn on,” or simply“On”—which are recognizable by resident VUI 36 of local device 14. Ifnothing has occurred to activate or start processing within local device14, method 100 repeats step 102. When an activating event does occur,and hence, processing is initiated within local device 14, method 100moves to step 104.

At step 104, local device 14 receives speech input from a user atmicrophone 20. This speech input—which may comprise audible expressionsof commands, instructions, directions, or requests spoken by the user—isforwarded to processing component 28. At step 106 processing component28 processes the speech input. Such processing may comprise preliminarysignal processing, which can include parameter extraction and/or speechrecognition. For parameter extraction, parameter extraction component 34transforms the speech input into a series of feature parameters, such asstandard cepstral coefficients, Fourier coefficients, LPC coefficients,or other parameters in the time or frequency domain. For speechrecognition, resident VUI 36 distinguishes speech using barge-incomponent 38, and may recognize speech at an elementary level (e.g., byperforming key-word searching), using speech recognition engine 40.

As speech input is processed, processing component 28 may generate oneor more responses. Such response can be a verbalized response which isgenerated by speech generation engine 42 and output to a user at speaker22. Alternatively, the response can be in the form of one or morecontrol signals, which are output from processing component 28 toprimary functionality component 19 for control thereof. Steps 104 and106 may be repeated multiple times for various speech input receivedfrom a user.

At step 108, processing component 28 determines whether processing ofspeech input locally at local device 14 is sufficient to address thecommands, instructions, directions, or requests from a user. If so,method 100 proceeds to step 120 where local device 14 takes action basedon the processing, for example, by replying to a user and/or controllingprimary functionality component 19. Otherwise, if local processing isnot sufficient, then at step 110, local device 14 establishes aconnection between itself and remote device 12, for example, viatelecommunications network 16 or local area network 18.

At step 112, local device 14 transmits data and/or speech input toremote system 12 for processing therein. Local device 14 at step 113then waits, for a predetermined period, for a reply or response fromremote system 12. At step 114, local device 14 determines whether atime-out has occurred—i.e., whether remote system 12 has failed to replywithin a predetermined amount of time allotted for response. A responsefrom remote system 12 may comprise data for producing an audio and/orvideo output to a user, and/or control signals for controlling localdevice 14 (especially, primary functionality component 19).

If it is determined at step 114 that remote system 12 has not repliedwithin the time-out period, local device 14 may terminate processing,and method 100 ends. Otherwise, if a time-out has not yet occurred, thenat step 116 processing component 28 determines whether a response hasbeen received from remote system 12. If no response has yet beenreceived from remote system 12, method 100 returns to step 113 wherelocal device 14 continues to wait. Local device 14 repeats steps 113,114, and 116 until either the time-out period has lapsed or,alternatively, a response has been received from remote system 12.

After a response has been received from remote system 12, then at step118 local device 14 may terminate the connection between itself andremote device 12. In one embodiment, if the connection comprises atoll-bearing public switched telephone network (PSTN) connection,termination can be automatic (e.g., after the lapse of a time-outperiod). In another embodiment, termination is user-activated; forexample, the user may enter a predetermined series of dual tone multiplefrequency (DTMF) signals at manual input device 24.

At step 120, local device 14 takes action based upon the response fromremote system 12. This may include outputting a reply message (audibleor visible) to the user and/or controlling the operation of primaryfunctionality component 19.

At step 122, local device 14 determines whether this interactive sessionwith a user should be ended. For example, in one embodiment, a user mayindicate his or her desire to end the session by ceasing to interactwith local device 14 for a predetermined (time-out) period, or byentering a predetermined series of dual tone multiple frequency (DTMF)signals at manual input device 24. If it is determined at step 122 thatthe interactive session should not be ended, then method 100 returns tostep 104 where local device 14 receives speech from a user. Otherwise,if it is determined that the session should be ended, method 100 ends.

Operation at Remote System

FIG. 6 is a flow diagram of an exemplary method 200 of operation forremote system 12, according to an embodiment of the present invention.

Method 200 begins at step 202 where remote system 12 awaits user inputfrom a local device 14. Such input—which may be received at transceiver50, LAN connector 52, or WAN connector 58—may specify a command,instruction, direction, or request from a user. The input can be in theform of data, such as a DTMF signal or speech. When remote system 12 hasreceived an input, such input is forwarded to processing component 54.

Processing component 54 then processes or operates upon the receivedinput. For example, assuming that the input is in the form of speech,echo cancellation component 64 of VUI 62 may remove echoes caused bytransmission delays or reflections, and barge-in component 66 may detectthe onset of human speech. Furthermore, at step 204, speech recognitionengine 70 of VUI 62 compares the command, instruction, direction, orrequest specified in the input against grammars which are contained ingrammar component 74. These grammars may specify certain words, phrases,and/or sentences which are to be recognized if spoken by a user.Alternatively, speech recognition engine 70 may compare the speech inputagainst one or more acoustic models contained in acoustic modelcomponent 73.

At step 206, processing component 62 determines whether there is a matchbetween the verbalized command, instruction, direction, or requestspoken by a user and a grammar (or acoustic model) recognizable byspeech recognition engine 70. If so, method 200 proceeds to step 224where remote system 12 responds to the recognized command, instruction,direction, or request, as further described below. On the other hand, ifit is determined at step 206 that there is no match (between a grammar(or acoustic model) and the user's spoken command, instruction,direction, or request), then at step 208 remote system 12 requests moreinput from a user. This can be accomplished, for example, by generatinga spoken request in speech generation engine 72 (using eithertext-to-speech component 76 or play-back component 78) and thenforwarding such request to local device 14 for output to the user.

When remote system 12 has received more spoken input from the user (attransceiver 50, LAN connector 52, or WAN connector 58), processingcomponent 54 again processes the received input (for example, using echocancellation component 64 and barge-in component 66). At step 210,speech recognition engine 70 compares the most recently received speechinput against the grammars of grammar component 74 (or the acousticmodels of acoustic model component 73).

At step 212, processing component 54 determines whether there is a matchbetween the additional input and the grammars (or the acoustic models).If there is a match, method 200 proceeds to step 224. Alternatively, ifthere is no match, then at step 214 processing component 54 determineswhether remote system 12 should again attempt to solicit speech inputfrom the user. In one embodiment, a predetermined number of attempts maybe provided for a user to input speech; a counter for keeping track ofthese attempts is reset each time method 200 performs step 202, whereinput speech is initially received. If it is determined that there areadditional attempts left, then method 200 returns to step 208 whereremote system 12 requests (via local device 14) more input from a user.

Otherwise, method 200 moves to step 216 where processing component 54generates a message directing the user to select from a list of commandsor requests which are recognizable by VUI 62. This message is forwardedto local device 14 for output to the user. For example, in oneembodiment, the list of commands or requests is displayed to a user ondisplay 26. Alternatively, the list can be spoken to the user viaspeaker 22.

In response to the message, the user may then select from the list byspeaking one or more of the commands or requests. This speech input isthen forwarded to remote system 12. At step 218, speech recognitionengine 70 of VUI 62 compares the speech input against the grammars (orthe acoustic models) contained therein.

At step 220, processing component 54 determines whether there is a matchbetween the additional input and the grammars (or the acoustic models).If there is a match, method 200 proceeds to step 224. Otherwise, ifthere is no match, then at step 222 processing component 54 determineswhether remote system 12 should again attempt to solicit speech inputfrom the user by having the user select from the list of recognizablecommands or requests. In one embodiment, a predetermined number ofattempts may be provided for a user to input speech in this way; acounter for keeping track of these attempts is reset each time method200 performs step 202, where input speech is initially received. If itis determined that there are additional attempts left, then method 200returns to step 216 where remote system 12 (via local device 14)requests that the user select from the list. Alternatively, if it isdetermined that no attempts are left (and hence, remote system 12 hasfailed to receive any speech input that it can recognize), method 200moves to step 226.

At step 224, remote system 12 responds to the command, instruction,direction or request from a user. Such response may include accessingthe Internet via LAN connector 58 to retrieve requested data orinformation. Furthermore, such response may include generating one ormore vocalized replies (for output to a user) or control signals (fordirecting or controlling local device 14).

At step 226, remote system 12 determines whether this session with localdevice 14 should be ended (for example, if a time-out period haslapsed). If not, method 200 returns to step 202 where remote system 12waits for another command, instruction, direction, or request from auser. Otherwise, if it is determined at step 216 that there should be anend to this session, method 200 ends.

In an alternative operation, rather than passively waiting for userinput from a local device 14 to initiate a session between remote system12 and the local device, remote system 12 actively triggers such asession. For example, in one embodiment, remote system 12 may activelymonitor stock prices on the Internet and initiate a session with arelevant local device 14 to inform a user when the price of a particularstock rises above, or falls below, a predetermined level.

Accordingly, as described herein, the present invention provides asystem and method for a distributed voice user interface (VUI) in whichremote system 12 cooperates with one or more local devices 14 to delivera sophisticated voice user interface at each of local devices 14.

CONCLUSION

Although particular embodiments of the present invention have been shownand described, it will be obvious to those skilled in the art thatchanges and modifications may be made without departing from the presentinvention in its broader aspects, and therefore, the appended claims areto encompass within their scope all such changes and modifications thatfall within the true scope of the present invention.

1. A voice integration platform for developing a voice interface to anexternal data system, the voice integration platform comprising: a voicegateway that enables voice interaction; a first set of service layersthat interact with the voice gateway; a second set of service layersthat facilitate interaction between the first set of service layers andthe external data system, the second set of service layers including adeveloper-accessible interface to content management software exposed bythe external data system, the content management software configured toexpose supplemental information regarding data in the external datasystem to a development platform of a specific voice interface; and oneor more computing devices that execute the voice gateway, the first setof service layers, and the second set of service layers; wherein thefirst and second sets of service layers and the supplemental informationregarding data in the external data system are used together to developthe specific voice interface that enables interaction via voice with theexternal data system by receiving a voice input from the voice gatewayat the first set of service layers and accessing data via the contentmanagement software of the external data system at the second set ofservice layers responsive to the voice input.
 2. The voice integrationplatform of claim 1, further comprising a tools service layer that worksin conjunction with the voice gateway to monitor performance of thespecific voice interface.
 3. The voice integration platform of claim 1,wherein the second set of service layers includes a content managementservice layer having a software module that enables the specific voiceinterface to parse a stream of audio content to define constituentportions, convert the parsed audio content to text files, and associatethe text files with metadata via interaction with the data system,thereby facilitating interaction of the specific voice interface withaudio content in the data system.
 4. The voice integration platform ofclaim 3, wherein the content management service layer provides atemplate for creation of dialogs in the specific voice interface,thereby facilitating voice interaction with the data system.
 5. Thevoice integration platform of claim 1, wherein the second set of servicelayers includes a personalization service layer having a software modulethat facilitates interaction of the specific voice interface withpersonalization content in the data system.
 6. The voice integrationplatform of claim 1, wherein the second set of service layers includesan integration service layer having an input/output layer that enablesthe specific voice interface and the data system to exchange data. 7.The voice integration platform of claim 1, wherein the first set ofservice layers includes an infrastructure service layer that includes adomain controller software component that manages and controls theorganization and storage of information into logically distinct storagecategories.
 8. The voice integration platform of claim 1, wherein thefirst set of service layers includes a personalized dialogs layer thatincorporates naturalized language concepts to present a realisticspecific voice interface.
 9. The voice integration platform of claim 1,wherein the first set of service layers includes an applications servicelayer that includes an email component that enables receiving andsending of email messages in voice format using the specific voiceinterface.
 10. The voice integration platform of claim 1, wherein thefirst set of service layers includes an applications service layer thatincludes a notification component that enables the handing offinformation from the specific voice interface to a local device.
 11. Thevoice integration platform of claim 1, wherein the voice gateway is adistributed voice interface system wherein the specific voice interfacecontrols the functioning of a respective local device through servicesand capabilities of a remote system.
 12. The voice integration platformof claim 1, wherein the one or more devices are server hardwarecomprising: a processor; a memory; and a network interface.
 13. Thevoice integration platform of claim 1, wherein the supplementalinformation regarding data in the external data system comprisesclassification of files stored within the external data system.
 14. Thevoice integration platform of claim 13, wherein the classification offiles stored within the external data system comprises meta tagdescriptors associated with the files stored within the external datasystem.
 15. A method of generating a voice interface to an external datasystem, comprising: implementing a voice gateway in the voice interfaceto receive a voice input; using a first set of service layers to developfunctional capabilities for the voice interface that facilitateinteraction with the voice gateway; and using a second set of servicelayers to develop integration capabilities for the voice interface thatfacilitate interaction between the functional capabilities and theexternal data system, the second set of service layers including adeveloper-accessible interface to content management software exposed bythe external data system, the content management software configured toexpose supplemental information regarding data in the external datasystem to a development platform of a specific voice interface; whereinthe functional capabilities and the integration capabilitiescooperatively form the specific voice interface that enables voiceinteraction via the content management software of the external datasystem.
 16. The method of claim 15, further comprising monitoring aperformance of the specific voice interface using a tools service layerthat works in conjunction with the voice gateway and the first andsecond sets of service layers.
 17. The method of claim 15, wherein thesecond set of service layers includes a content management service layerand further comprising using the content management service layer toprovide a template for creation of dialogs in the specific voiceinterface, thereby facilitating voice interaction with the data system.18. The method of claim 15, further comprising facilitating interactionof the specific voice interface with personalization content in the datasystem via a personalization service layer that is included in thesecond set of service layers.
 19. The method of claim 15, furthercomprising allowing the specific voice interface and the data system toexchange data via an integration service layer having an input/outputlayer, the integration service layer being part of the second set ofservice layers.
 20. The method of claim 15, further comprising enablingan infrastructure service layer that includes a domain controllersoftware component to manage and control organization and storage ofinformation into logically distinct storage categories, theinfrastructure service layer being part of the first set of servicelayers.
 21. The method of claim 15, further comprising incorporatingnaturalized language concepts to present a realistic specific voiceinterface via a personalized dialogs layer, the personalized dialogslayer being part of the first set of service layers.
 22. The method ofclaim 15, further comprising enabling an applications service layer thatincludes an email component that receives and sends email messages invoice format, the applications service layer being part of the first setof service layers.
 23. The method of claim 15, further comprisingenabling an applications service layer that includes a notificationcomponent for handing off information from the voice interface to alocal device, the applications service layer being part of the first setof service layers.
 24. The method of claim 15, further comprisingimplementing a distributed voice gateway wherein the specific voiceinterface controls the functioning of a respective local device throughservices and capabilities of a remote system.
 25. The method of claim15, wherein the supplemental information regarding data in the externaldata system comprises classification of files stored within the externaldata system.
 26. The method of claim 25, wherein the classification offiles stored within the external data system comprises meta tagdescriptors associated with the files stored within the external datasystem.
 27. A computer-readable medium having stored thereoncomputer-executable instructions, execution of which, by a computingdevice, causes the computing device to perform operations for generatinga voice interface to an external data system, the operations comprising:implementing a voice gateway in the voice interface to receive a voiceinput; using a first set of service layers to develop functionalcapabilities for the voice interface that facilitate interaction withthe voice gateway; and using a second set of service layers to developintegration capabilities for the voice interface that facilitateinteraction between the functional capabilities and the external datasystem, the second set of service layers including adeveloper-accessible interface to content management software exposed bythe external data system, the content management software configured toexpose supplemental information regarding data in the external datasystem to a development platform of a specific voice interface; whereinthe functional capabilities and the integration capabilitiescooperatively generate the specific voice interface that enables voiceinteraction via the content management software of the external datasystem.
 28. A voice integration platform for developing a voiceinterface to an external data system, the voice integration platformcomprising: voice gateway processor means to enable voice interaction;first set of service layers means to interact with the voice gatewaymeans; and second set of service layers means to facilitate interactionbetween the first set of service layer means and the external datasystem, the second set of service layers means including adeveloper-accessible interface to content management software exposed bythe external data system, the content management software configured toexpose supplemental information regarding data in the external datasystem to a development platform of a specific voice interface; whereinthe first set of service layer means and second set of service layermeans are used together to develop the specific voice interface thatenables voice interaction via the content management software of theexternal data system.
 29. A voice integration platform for developing avoice interface to an external data system, the voice integrationplatform comprising: a voice gateway that enables voice interaction; afirst set of service layers that interact with the voice gateway; and asecond set of service layers that facilitate interaction between thefirst set of service layers and the external data system, the externaldata system comprising a content management software having a servicemodule to parse a steam of audio content to define constituent portionsbased on association with one of a plurality of logically distinctstorage categories and selectively storing the portions of the audiocontent stream in the one of the plurality of logically distinct storagecategories in the external data system, wherein the second set ofservice layers includes a developer-accessible interface to the contentmanagement software; wherein the first and second sets of service layersare used together to develop a specific voice interface that enablesvoice interaction via the content management software of the externaldata system, the specific voice interface to allow selective access, viathe voice gateway, of one of the stored portions of the audio contentstream for playback by traversing one or more of the plurality oflogically distinct storage categories.